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Reading comprehension tests of ten 'take the form. of reading pas- 
sages followed by multiple choice questions about the passages. Students' 
answers to the questions are taken as measures of their reading compre- 
hension ability. One potential weakness in such a measure of reading ' 
comprehensionjs that comprehension questions may be passage independent. 
That is, students. may be able to score reasonably well on comprehension 
questions without having read the passages intended to precede them. '. 
Preston (1964) found that col 1 ege students were able to-score signifi- 

_cantly:higher_than chance on the. Cooperative En ql ish LTest _when deprjved_ 

■ ■] " ■ - — / ■■ . ■ ■ " ~ 

of the passages to which the test questions were supposed to relate. 
Weaver and Bickley (1967) found that college sophomores deprived of read- • 
ing passages correctly -answered 67 percent as many comprehension ques-. 
tions as sophomores who read the passages. Tuinman .(1973-1974) , study- 
ing pe^ormance by fourth-, fifth-, and sixth-grade subjects on five 
highly- regarded and widely used reading tests, found that subjects de- 
prived of the reading passages scored h^^gher than chance on the- compre- 
hension questions. 



*Paper presented at the Twenty-Eighth Annual Meeting of the 
National Reading Conference, St. Petersburg, Florida, December 2, 1978. 



. Cyrus F. Smith, J r 
-2- Richard D. Western 

This potential weakness in traditional multiple choice measures J 
of comprehension is important in at least' two respects. Wherever it • 
proves to be an actual weakness, it contributes to the argument that 
performance on multiple choice tests is an inadequate measure of "reading 
comprehension. , Whatever reading -comprehension is, a test that invol ves' 
-no reading (except for reading of the test questions) cannot be taken as 
a measure of it. Moreover,. to the extent that multiple choice tests : 

prove to be passage independent, serious questions arise about the prac- 

■• ■ ' "jf^ ■ ■ ■ ■ ■ ^ ' • . 

tice of using scores from such tests as measures of proficiency upon 

which grade level promotion, or graduation, may be based. School sys- 
tems across the country are under-increasing pressure now to make such 
uses of scores derived, from multiple choice tests. 

One such test is the Stanford Test of Academic Skills (hereafter, . 
TASK). In its Manual it is represented as a test of reading comprehension 
(p. 7) suitable for use as a measure of proficiency in the basic skills 
(p.. 12). At least one large urban school system in Wisconsin has decided 
to use it to determine which of its students will be eligible for an 
academic high school diploma. The purpose of the present study is to 
assess the Stanford Test of Academic .Skills Level 1 , Fonns A and B, for 
passage independence. . ' 

• Method- 

» ' .I- . ^ • 

' . . . • ^ 

Subjects for the study were in? tenth graders randomly selected 
from an academically heterogeneous population of 489 tenth graders in one 
of two high schools in a middl e-size Wisconsin city, .X - 
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Materials for the study were developed from TASK Level 1 , Form 
A (questions 1-42), and Form B (questions 1-42). Each subject received 
a package of materials consisting of two parts: (1) a pas sage- out com- ' 
ponent (either Form A or Form B)r- that is, the publisher's questions with- 
out the reading passages; and, (2) a passage- in component" (either Form A 
or Form Bj—that is, the publisher's reading passages followed by ques- 
.tions. Forty-nine -subjects received the 42 comprehension items for. Form ' 
A ( passage-out condition) plus the reading passages and the comprehension 
items for Form B ( passage-in condition). Fifty-two subjects received' 
Form B passage-out plus Form A passage-in . . 0 

• The experimenters administered TASK, Level 1, Forms A and B . (with 
alterations as described above), inJOctober, 1977. Following roughly the 
example described by Tuinman (1.973-74) , they used standard directions as 
follows: " 

a- An announcement thai: the .tests were being used to get informa- 

, . tipn on the tests, not on the. subjects. 

b. An explanation of the two-fold character of the test: the 

-f-i-rs-t— h^a-lf a test with questions only; the-recon d^ a tes fuf ■ — : 

the usual sort with passages as well as tiijestions.. 

c. A statement that results would not be reported to teachers or 
to other school officials. 

d. A plea for cooperation. 

\ The administration of the test under, the passage-out and passage- 

■ — ^- \ • • • , -. * "■• { 

iji conditions lasted from 35-50 minutes* All 101 subjects completed their 

tasks in\the time allptted.'^ / . r ^ 

ERIC - / 4 
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■_ Analysts and Results 

■The study attempted,«to determine whether subjects, as a group, 
could achieve raw scores greater than, the chance score. Each form of 
the Stanford Test of Academic Skiiu, Level 1, contains 42 multiple choice 
items which are based upon reading passages. Since each item has four' 
cho.tces, the expected chance scare is 10.5' 

Of the 49 subjects who took Level 1, Form A, in the passage-out 
condition, 91.8 percent exceeded the- chance mean score. The- observed 
mean sco.re- for this group was 15.9. Analysis. by a one-tailed t-test, ' 
t=2. 269, shows this mean to: be significantly greater (p<0.025) than the 
expected chance mean. Fifty-two subjects took Level 1 , Form B, in the ' 
passage-out Condition. One hundred percent exceeded the expected chance 
mean score. The observed mean score for this gr-oup was 18.0. Analysis by 
a one-tailed t-test, t=2.048, showsthis mean to be significantly greater 
(p <0.025) than the expected chance mean score. : 

TABLE 1 .—Summary of Results: Passage. Out 
Condition, Stanford Test of Academic Skills . 
L - Level 1 , Forms A and B ' 



' TASK TASK 

Form A Form B 

- Passage^Out Passage-Out 

^ 15.9 18.0 

Chance X 10.5 10.5 

n 49 ,.52 

- 2.379 3.662 
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Calculations based upon a normal! approximation to the binominal 
distribution (where p=. 25 and n=42) were conducted for each group of 
scores. According to these calculations, only 1 percent of the sample, 
was expected to achieve a raw score. higher than 17.05. However, 34:7 per- 
cent of the Form A sample and 51.9 percent of the Form B sample exceeded 
the expected score of 17.05. These data are presented in Table 2. 

TABLE 2. — Percentages of Subjects Who Beat Chance ^ tanford Test 
of Academic Skills, Level 1, Form A and B 

TASK Form A,. TASK Form B, 

% of Passage Out. Passage Out. ' 

Chance Sample % of Sample %. of Sample 

Raw Expected to . Exceeding Chance Exceeding Chance 

- Score Exceed Chance (n=49) (n=52) 

10.50 50.0 91 :8-. 100.0 

. ' 12.41 . 25.0 73.5 ^ 90.1 

14.09 10.0 51 .0 69;2 

.,___.__J_6....0T 2.5 ■ , ' 38.8 . ' ' 57.7 

17.05 1.0 34.7 51.9 



Finally, Pearson-Product Moment correlations were calculated" 
upon. those scores achieved by subjects in either the Form A passage out— 
Form B pas sage- in condition or the Form B passage out— Form... A passage- * 
jjrcondition. Significant correlation coefficient (p<. 01 ) of .5753 and 
.7457 were obtained for the. former arid latter conditions respec tively.- 
These data jare presented in TabTe. 3/ . 
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TABLE 3. --Pearson-Product Moment Correlations of Passage Out and 
Passage In Conditions, Stanford Test of Academic Skills , Level 1 
^ ., Forms A and B 



Condition 


n 


X 


s.d. 


r . 


TASK Form A, Passage Out 


49 


15.9 . 


4.44 


.5753** 


TASK Form B, Passage In 


49 


,27.2 


/ 9.09 ■ 




TASK FORM B, Passage Out 


52 


18.0 


4.39 


.7457** 


TASK Form A, Passage In 


52 


28.5 


.10.87 





**p<.01 



Discussion " 

This study is generally consistent with the findings of Preston 
{1964), Weaver (1967), and Tuinman (1973-1974). Taken together, these 
^studies show that subjects have been able to perform quite well in a pass- 
age-out condition on a variety of standardized tests of reading comprehen- 
sion. this information, coupled with the enactment of reading proficiency 
requirements by various school systems, gives new point to an old concern- 
regarding valid measurement of reading compirehensi on.. Traditionally, read- 
ing comprehension has been defined utilizing the passage-in model,. That 
is to say, subjects are confronted with a reading text and comprehension 
of that text is assessed by multiple choice questions. It might.be argued 
that .reading of the sort done under the passage-out condition also involves' 
reading comprehension, since subjects do at least read the questions.. How- 
ever, the fallacy of this line of reasoning, at least as applied to pro-- ' 
ficiency testing, can be seen when one considers the different roles of 
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background knowledge, experience, thinking ability, and the like in the ' 
two situations. . „ 

In passage- in type reading, background knowledge, experience, and 
so forth are probatDly necessary conditions (Tuinman, 1973-1974) for compre- 
hension in that they help the reader interpret the text. In passage-ou t 
reading, however, background knowledge, experience, thinking ability, and 
so forth are sufficient conditions for comprehension in that the questions 
send the reader not to an examination of the text but away from the text 
to an examination of matters exclusive of it. Inconsequence, the word 
"reading" is apparently used in two radically different senses depending 
upon whether the passage-in or passage-out condition obtains. The. corre- 
lation coefficients of .5753 and ,7457 (see Table. 3) calculated f rom the • 
comprehension scores of subjects, in the passage-in and passage-out condi;- 
tions provide some evi"dence for this contention. These coefficients, when 
squared, show that the passage-in reading condition accounts for only 33 
percent and 55 percent of the variance in the com prghen£ion scores -re- 
spectively for Form A and B of the Stanford Test of Academic Skills , Level 1. 

Ppnpl p d pmaDdlngJ±esls-of— read j nglpro^ic-iency as tests of basic 

reading skills almost certainly mean reading in the passage- in sense. For 
measures of such proficiency, therefore, a test on which results may be 
seriously confounded with passage-out reading ability is clearly inade- 
quate . - It i s - i nadequate i n -tha t-i t-does - not-"separate -ou t— thos e.-^lemen ts 
^that are unique to reading, and. those that depend but little- on reading" 

(Calfee, 1976:42). A test that did make such a separation might be, as 

• . . ■ . '■ ■ ■ ■ ■ ■ ■ ■ 

Calfee suggests, an entirely new sort of test. 
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Continued academic, analysis 'Of "compreho.nsion" and declining stu- 
dent populations make the present a likely time for school districts to 
begin an exploration for new means of measurement. Unfortunately, the mood 
of. concern being, spread by the back-to-basics movement apparently is forc- 
ing school districts in" the, direction of expediency, not exploration. This 
probably means that use. of passage independent tests will continue In that 
tests of this type create the illusion that reading proficiency is -being 
assessed. . , • 
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